IDF term weighting and IR research lessons

Robertson comments on the theoretical status of IDF term weighting. Its history illustrates how ideas develop in a specific research context, in theory/experiment interaction, and in operational practice. It is an honour to have the small proposal for term weighting that I published more than thirty years ago (Sparck Jones 1972) the subject of Stephen Robertson’s paper (Robertson 2004). I would like to comment on some points that I see as suggesting lessons for information retrieval research. First, the context that prompted the proposal. The proposal came from trying to explain why earlier ideas about how to do automatic indexing did not work. They were plausible in themselves, but had quite different objectives. My previous research had concentrated on automatic methods for constructing term classifications intended, by analogy with manual thesauri, as recall-promoting devices. Classes were based on term cooccurrences in documents, following the generic statistical approach to retrieval initially suggested by Luhn, and applied within the coordination level matching framework. But these classifications, on Cleverdon’s Cranfield data and using the test and evaluation methods that he and Salton had established, did not deliver the predicted improvements in retrieval performance. The best performance was obtained with (necessarily) small groups of very similar terms. Trying to understand what was happening in detail showed that terms that occurred in many documents dominated the classes. Thus anything that increased their matching potential, as term substitution did, would inevitably retrieve non-relevant rather more than relevant documents. However these frequent terms were also common in requests, and simply removing them, as advocated by Svenonius, could have a damaging effect on performance. The natural implication was therefore that less frequent terms should be grouped but more frequent ones should be confined to singleton classes. This could give better performance than terms alone, but not for all test collections. What all this suggested was that it might be more profitable to concentrate on the frequency behaviour of terms, and forget about classes. More specifically, it led to the idea that all terms should be allowed to match but the value of matches on frequent terms should be lower than that for non-frequent terms. Roger Needham, who had earlier worked on statistically-based methods of indexing and retrieval, was easily able as a mathematician to suggest an appropriate simple formula that smoothly damped down frequency and was shown to work reliably and usefully for different collections. My proposal for weighting was thus a direct response to the results of the kind of systematic retrieval testing that Cleverdon did so much to establish. It was also, when compared with Salton’s work on automatic indexing, a product of subtly different data conditions. Salton